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Abstract. In this paper, we consider the task of answering linear queries 
under the constraint of differential privacy. This is a general and well- 
studied class of queries that captures other commonly studied classes, 
including predicate queries and histogram queries. We show that the 
accuracy to which a set of linear queries can be answered is closely re- 
lated to its fat-shattering dimension, a property that characterizes the 
learnability of real- valued functions in the agnostic-learning setting. 



1 Introduction 

The administrator of a database consisting of sensitive, but valuable 
information faces two conflicting objectives. Because the data is valuable, 
she would like to make statistical information about it available to the 
public. However, because the data is sensitive, she must take care not to 
release information that exposes the data of any particular individual in 
the data set. The central question in the field of private data analysis is 
how these two objectives can be traded off, and more specifically, how 
many queries of what type can be answered to given degrees of accuracy, 
while still preserving privacy. 

Recent work on differential privacy provides a mathematical framework 
to reason about such questions. Informally, a probabilistic function / 
from a database D to some range TZ is a-differentially private if adding 
or removing a single individual from the dataset does not change the 
probability that f{D) = r for any outcome r G 7?. by more than an e" 
factor. The intuition behind this definition is that an individual's privacy 
should not be considered to have been violated by some event r, if r would 
have been almost as likely to occur even without the individual's data. 
In this paper, we consider databases D which are real valued vectors, and 
the class of queries that we consider correspond to linear combinations of 
the entries of D. Formally, we consider databases D £ W^, and queries of 
the form q £ [0, 1]". The answer to query q on database D is simply the 
dot-product of the two vectors: q{D) = q - D. This model has previously 
been considered ( |DN03IDMT07IDY08IHT10j ) . and generalizes the class 
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of count queries or predicate queries, which has also been well studied 
f |DMNS06IBLR08IDNRRV09IRR10IUVT0] ) . 

The fat- shattering dimension (FSD) of a class of real-valued functions 
C over some domain is a generalization of the Vapnik-Chervonenkis di- 
mension, and characterizes a distribution-free convergence property of 
the mean value of each / £ C to its expectation. The fat-shattering 
dimension of a class of functions C is known to characterize the sam- 
ple complexity necessary to PAC learn C in the agnostic framework 
[XBC H97 BLW94,: that is, ignoring computation, the sample complexity 
that is both necessary and sufficient to learn C in the agnostic framework 
is polynomially related to the fat-shattering dimension of C. 
Our main result is a similar information theoretic characterization of the 
magnitude of the noise that must be added to the answer to each query 
in some class C in terms of the fat-shattering dimension of C, FSD(C). 
We show polynomially related information theoretic upper and lower 
bounds on the noise that must be added to each query in C in terms of 
FSD(C). This generalizes the results of |BLR08| to linear queries, and to 
our knowledge gives the first analysis of generic linear queries using some 
parameter other than their cardinality. This yields the first mechanism 
capable of answering a possibly infinite set of generic linear queries, and 
the first non-trivial lower bound for infinite classes of non-boolean linear 
queries. As a consequence, we extend results of Kasiviswanathan et al. 
and Blum et al. |KLN RS08 BLRO8 relating the sample complexity nec- 
essary for agnostic PAC learning and private agnostic PAC learning from 
classes of boolean valued functions to classes of real valued functions. 

1.1 Related Work and Our Results 

Dinur and Nissim studied the special case of linear queries for which both 
the database and the query are elements of the boolean hypercube {0, 1}" 
[DN03 . Even in this special case, they showed that there cannot be any 
private mechanism that answers n queries with error o[y/n), because 
an adversary could use any such mechanism to reconstruct a 1 — o(l) 
fraction of the original database, a condition which they called blatant 
non-privacy. This result was strengthened by several subsequent papers 
[DMT07 DY08 KRSUlO;. 

Beimel ct al. consider the class of basis vectors on the boolean hypercube, 
and show that even though this class has a constant VC-dimension (and 
hence fat-shattering dimension), it requires a superconstant number of 
samples for useful private release [BKNIO]. Specifically, they show that 
the logn factor which appears in the upper bound in this paper and in 
|BLR08] . but not in the lower bound, is in fact necessary in some cases. 
Dwork et al. gave the original definition of differential privacy, as well 
as the Laplace mechanism, which is capable of answering any k "low 
sensitivity" queries (including linear queries) up to error 0(k). A more 
refined analysis of the relationship between the laplace mechanism and 
function sensitivity was later given by [NRS07) . 

In a different setting, Blum Ligett and Roth considered the question of 
answering predicate queries over a database drawn from some domain 
X (BLR08) . This can be viewed as a special case of linear queries in 
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which the queries are restricted to lie on the boolean hypercube, and the 
database must be integer valued: D G Z". They give a mechanism for 
answering every query in some class C with noise that depends linearly 
on the VC-dimension of the class of queries. This is a quantity that is at 
most log \ C\ for finite classes C, and can be finite even for infinite classes. 
Roth and Roughgarden later gave a mechanism which achieved similar 
bounds in the online model, in which the mechanism does not know 
the set of queries that must be answered ahead of time, and instead 
must answer them as they arrive [RRlOj . We generalize the technique 
of [BLR08IRR10) to apply to general linear queries. VC-dimension is no 
longer an appropriate measure of query complexity in this setting, but 
we show that a quantity known as Fat-Shattering dimension plays an 
analogous role. 

Dwork et al. [DNRRV09j also gave upper and lower bounds for pred- 
icate queries, which are incomparable to the bounds of BLR08 RRIO] 
(and those presented in this paper). The upper bounds of DNRRVQ9] 
are for an approximate form of difi'erential privacy, and have a better 
dependence on a, but a worse dependence on k. Their lower bounds are 
computational, whereas the lower bounds presented in this paper are 
information theoretic. 

Hardt and Talwar [HTlOj give matching upper and lower bounds on the 
noise that must be added for a-differential privacy when answering k < n 
linear queries of roug 

hiy g,( V^iog(n/fc) -^^ contrast, we prove bounds in 
terms of different parameters, and can handle arbitrarily (even infinitely) 
large values of k. For finite sets of k queries, our mechanism adds noise 
roughly 

O ■ ^ i°skiogn y/3^ j^g^g ^j^^^ ^Qj. g^j^g settings of the parame- 

ters, this is significantly less noise than the bounds of [HTIO) : specifically, 
for k > i7(| |_D| I j^^). To achieve low relative error rj (i.e. error e — 7)||D||i), 
our mechanism requires only that ||-D||i be poly-logarithmic in k, rather 
than polynomial in k. For infinite classes of queries |C|, the logfc in our 
bound can be replaced with the fat shattering dimension of the class C. 
We also show a lower bound in terms of the fat shattering dimension of 
the class C, which is the first non-trivial lower bound for infinite classes 
of non-boolean linear queries. 



2 Preliminaries 

A database is some vector D G R", and a query is some vector q £ 
[0, 1]". We write that the evaluation of g on _D is q{D) = q - D. We write 
\\D\\i = X^iLi ^« to denote the £i norm of D, and note that for any 
query q, q{D) £ [0, ||-D||i]. We let C denote a (possibly infinite) class 
of queries. We are interested in mechanisms that are able to provide 
answers for each G C so that the maximum error, defined to be 
maxigc \<li{D) — ai\ is as small as possible. Without loss of generality, 
we restrict our attention to mechanisms which actually output some 
synthetic database; mechanisms with range TZ = R". That is, if our 
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mechanism outputs some synthetic database D' , we take Ui to be qi{D') 
for each i0 

We formalize our notion of utihty and relative utility for a randomized 
mechanism M: 

Definition 1 (Usefulness and Relative Usefulness). A mechanism 
M : R" — >■ R" is (e, 5 )-useful with respect to a class of queries C if with 
probability at least 1 — 5 (over the internal coins of the mechanism), it 
outputs a synthetic database D' such that: 

sup |gi(-D) - qi{D')\ < e 
nec 

For < rj < 1, M is (77, 5)-relatively useful with respect to C for databases 
of size s if it IS (ri\\D\\i, 5)-useful with respect to C for all input databases 
D with \\D\\i > s. 

That is, useful mechanisms should have low error for each query in C. 
We now define differential privacy: 

Definition 2 (Differential Privacy | DMNS06'j ) ■ A mechanism M : 
Wl — > R4I IS a -differentially private, if for any two databases D\,D2 
such that \\Di - D2\\i < I, and for any S C Ri;:.- 

Pr[M{Di) e S]< e" Pr[M{D2) £ S] 

The standard notion of differential privacy need only hold for mecha- 
nisms defined over integer valued databases Di , D2 £ N" , which is a 
weaker condition. Our upper bounds will hold for the stronger notion 
of differential privacy, and our lower bounds for the weaker notion. A 
useful observation is that arbitrary (database independent) functions of 
differentially private mechanisms are also differentially private: 

Fact 1 // M -.W^^Wl IS a- differentially private, and if f : 

IS a (possibly randomized) function, then f{M) is a-differentially private. 

2.1 Fat Shattering Dimension 

_Fat-s/iattenn(/-dimension is a combinatorial property describing classes 
of functions of the form / : X — >■ [0, 1] for some domain X. It is a gen- 
eralization of the Vapnik-Chervonenkis-dimension, which is a property 
only of classes of boolean valued functions of the form / : X — >■ {0, 1}. 
In this section, we generalize these concepts slightly to classes of linear 
queries, where we view our linear queries as linear combinations of func- 
tions / : X — >■ [0, 1], where we let X be the set of standard basis vectors 
of R". 

Let B = {ei}"^i denote the set of n standard basis vectors of R" (e^ is the 
vector with a 1 in the i'th coordinate, and a in all other coordinates). 

^ This is without loss of generality, because given a different representation for each 
answer Oi to error e, it is possible to compute a synthetic database D' with error at 
most 2e using the linear program of [DNRRV09j . 



Differential Privacy and the Fat-Shattering Dimension of Linear Queries 



5 



For any S C _B of size \S\ — d, we say that S is 7-shattered by C if there 
exists a vector r G [0, 1]'' such that for every b £ {0, 1}'', there exists a 
query qt £ C such that for each G S: 



Note that since the range of each query is [0, 1], 7 can range from to 



Definition 3 (Fat Sliattering Dimension | BLW94IK"S94] '). The 7- 

f at- shattering dimension of a class of linear queries C is: 

FSDj{C) = max{d G N : C 7 - shatters some S Q B with \S\ = d} 

In the special case when 7 = = 1/2 for all i, note that the fat shattering 
dimension of a class of boolean valued functions is equal to its VC- 
dimension. 

For finite classes C, we will let fc = |C| denote the cardinality of C. 
The following observation follows immediately from the definition of fat- 
shattering dimension: 

Observation 1 For finite classes C , FSD-y (C) < log k for all 7 > 0, 

where k = |C| . 

3 Lower Bound 

In this section, we show that any a-differentially private mechanism that 
answers every linear query in some class C must add noise at least linear 
in the fat-shattering dimension of C at any scale. The bound that we 
prove in this section is in terms of the privacy parameter a and the fat 
shattering dimension of the class. It differs from the upper bound proved 
in the next section by several important parameters, which include a 
logn term and a term depending on the size of the database. Beimel 
et al. [BKNIO) have shown that the logn term in the upper bound is 
necessary in some contexts. The database that we construct in our lower 
bound is of size 0(7-FSD^(C)). Therefore, in order to prove a nontrivial 
lower bound on the relative error achievable by a private mechanism, it 
would be necessary to remove a factor of 7 from our current bound. This 
is possible in the context of VC-dimension, and we conjecture that it 
should also be possible for a bound in terms of fat-shattering dimension, 
and is merely a limitation of our techniques as present. The problem of 
proving a tight lower bound encapsulating all of the relevant parameters 
remains an interesting open question. We now proceed with the lower 
bound; 

Theorem 2. For any 5 bounded away from 1 by a constant, let M be 
a mechanism M that is (e, S) useful with with respect to some class of 
linear queries C. If M preserves a- differential privacy, then 
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We begin with some preliminaries which allow us to prove some useful 
lemmas: 

Given some class of linear queries C and any 7 > 0, let 5 C B be a 
collection of basis- vectors of size FSD-y(C) that are 7-shattered by C, 

and let r G [0, be the corresponding vector as in the definition 

of fat-shattering dimension. We now partition S into I/7 pieces. For each 
j €{!,..., 1/7}, let: 

= {ei e S : (i - 1) • 7 < ri < i • 7} 

Since the sets {Sj} partition S, By the pigeon-hole principle, there exists 
some f such that \S'" \ > 7; |S| = 7 ■ FSD-, (C). Let d = {S'" |. 
We consider subsets T C of size |T| = d/2. For each such subset, we 
consider the database Dt = X^e eT*^*- ^"^^ ^ {0,1}'* bo the vector 
guaranteed by the definition of fat shattering dimension such that: 

^ I 1, ei € T; 
' \ 0, otherwise. 

Let qt € C he the query that corresponds to as in the definition of 

fat shattering dimension, and let C^j* = {qr : T C S-' ,\T\ = d/2}. 
We first show that each function qt takes its highest value on Dt and 
cannot take large values on databases Dt' for sets T' that differ signifi- 
cantly from T. 

Lemma 1. For all qr £ Cgr and for all T' C S^' with \T'\ = d/2: 
qriDr) - qriDr') > | ■ |r A T'| 

Proof. 

qriDr) - qriDr') = ^ qriei) - ^ qriei) 

= qriei) -qriei)] + Y l^ri^i) 

- Y 1'^^^^^ 




>27-|r\T'|-| max ri- min ri]-\T\T'\ 

~ \ier'\r ier\r' J 

>7.|r\r'| 



where the last inequality follows from the fact that T,T' C which 
was constructed such that: 



max n — min rj 1 < 7 

\ieSi' ieSi' } 

holds. Observing that IT A r'| = 2|T \ T'] completes the proof. 
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With this lemma, we are ready to prove the main technical lemma for 
our lower bound: 

Lemma 2. For any S bounded away from 1 by a constant, let M be an 
{e, 5)-useful mechanism with respect to class C. Given as input M{Dt), 
where Dt is an unknown private database for some T <Z S-' with \T\ = 
d/2, with constant probability 1 — 5, there is a procedure to reconstruct a 
new database Dt' such that \T AT*\ < 

Proof. Suppose that mechanism M is (e, S) useful with respect to C for 
some constant 5 bounded away from 1. Then by definition, with constant 
probability, given input Dt, it outputs some database D' such that for 
all Qi e C, \q,{DT) - qr{D')\ < e. For each T' C 5^' with |r'| = d/2 let: 

v{T')=qT>iDT')-qT'iD') 

and let T* — argmin^, v{T'). Therefore, we have: 

v{T*) < v{T) = qriDr) - qriD') < e (1) 

where the last inequality follows from the usefulness of the mechanism. 
We also have: 

v{T*) =qT'{DT')-qT'{D') 

> qr' {Dt' ) — QT* (Dt) — £ 

> ^ • IT AT*! - e 



where the first inequality follows from the usefulness of the mechanism, 
and the second inequality follows from lemma [T] Combining this with 
equation [1] we get: 

4e 

|T AT*| < — 

7 

We are now ready to prove the lower bound: 

Proof (Proof of Theorem). Let T C S^' with |T| = d/2 be some ran- 
domly selected subset. Let Dt ~ X^e gT corresponding database. 
By lemma[51 given M{Dt, e), with probability 1 — 5 there is a procedure 
P to reconstruct a database Dt* such that |T Ar*| < 4e/7. Throughout 
the rest of the argument, we assume that this event occurs. Let j: G T be 
an element selected from T uniformly at random, and let j/ £ S\T be an 
element selected from S\T uniformly at random. Let T' = T\x\J {j/}. 
Observe that: 

p,MP(«p„),i>;!M.,_Ji. 

Pr|«P(A/(C,..0)l<^.:i!5 
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Since j| Dt — I^t' 111 — 2, we have by the definition of a-differential privacy 
and fact[T] 

^„ ^ Pr[x e PiM{DT,e))] 



PT[x&P{M{DT',e))] 

4e 



4fe 

7 ■ _ 
4e 



Solving for e, we find that: 

7 • d 



Since this holds for all choices of 7, the claim follows from the fact that 
d > 7FSD^(C). 



4 Upper Bound 

We now show that (ignoring the other important parameters), it is suffi- 
cient to add noise linear in the fat shattering dimension of C to simulta- 
neously guarantee usefulness with respect to C and differential privacy. 
Unlike our lower bound which was not quite strong enough to state in 
terms of relative error, our upper bound is most naturally stated as a 
bound on relative error. 

We make use of a theorem of Bartlett and Long [BL95) (improving a 
bound of Alon et al. [ABCH97] ) concerning the rate of convergence of 
uniform Glivenko-Cantelli classes with respect to their fat-shattering di- 
mension. 



Theorem 3 ( |BL95| Theorem 9). Let C be a class of functions from 
some domain X into [0, 1]. Then for all distributions P over X and for 
all rj,5 > 0; 



Pr 



sup 

/ec 



1 

-^/(a:0-E..p[/(^)] >77 



< 5 



where {xi}YLi are m independent draws from P and 



O 



In 



In- 



where d. 



17/5 



F5i?,/5(C). 



We use this theorem to prove the following useful corollary: 

Corollary 1. Let C be a class of linear functions with coefficients in 
[0, 1] from to R. For any database D £ R" , there is a database D' G 
N" with 

|D'||,=0('%.log^f^~ 



Differential Privacy and the Fat-Shattering Dimension of Linear Queries 



9 



such that for each q G C, 
where = FSD^/^{C) . 



\D 



< v\\D\\i 



Proof. Let B = {ei}"^i denote the set of n standard basis vectors over 
R". Let P_D be the probabihty distribution over B that places probability 
Di/||-D||i on ei. Note that for any q £ C: 



D 



1 " 



Let xi, 



, Xm he m = O 



/5 



In^ 



In 2 



D\\i 

independent draws 



from Pd, and let D' = YJHi ^i- Then: 

n m 

q{D') ^Y.1^D'^^^) =Y.1^' 



By lemma [3l we have 
q{D') q{D) 



Pr 



i^lii 



Pr 



<i 
- 2 



In particular, there exists some D' £ N" with ||-D'||i ~ m that satisfies 
< r}. Multiplying through by ||-D||i gives the desired 



I HD'lll 

bound. 



Armed with Corollary [T] we may now proceed to instantiate the expo- 
nential mechanism over a sparse domain, analogously to the instantiation 
of the exponential mechanism in BLR08 . 

Definition 4 (The Exponential Meciianism |MT07p . Let V he 

some domain, and let s : x O — )■ R 6e some quality score mapping 
database/domain- element pairs to some real value. Let 



As > max 



sup 



\s{Di,r)^s{D2,r)\ 



Di,026B 



l-Dl--D2lll<l 



be an upper bound on the £1 sensitivity of s. The exponential mechanism 
defined with respect to domainT) and score s is the probability distribution 
(parameterized by the private database D) which outputs each r G 2? with 
probability proportional to: 



' exp 



s(D, 



2As 



Theorem 4 (McSherry and Talwar |MT07] ). The exponential mech- 
anism preserves a- differential privacy. 
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We let m — O (^-^ ■ log'^ (■^))' ^^'^ define the domain of our instanti- 
ation of the exponential mechanism to be: 

V ^ {D' e N" : \\D'\\i = m} 

We note that I'D] — n™. Finally, we sample each D' £1) with probability 
proportional to: 

D' ~ exp 1 ^ L (2) 



and output the database Dout = Observe that for any two 

databases D\,D2 such that IIDi — O2II1 < 1 we have: 



i^(^l) - ^SlT ■ - - ■ q{D')\ < 

Di-D2\\i + 



"!^i|!i-i!^2!li 



1+1 

m 

Therefore, the distribution defined in equation [2] is a valid instantiation 
of the exponential mechanism, and by |MT07) preserves a-differential 
privacy. It remains to show that the above instantiation of the exponen- 
tial mechanism yields a useful mechanism with low error. In particular, 
it gives us a relatively useful mechanisms with respect to classes C for 
databases that have size linear in the fat shattering dimension of C, or 
only logarithmic in |C| for finite classes C. This is in contrast to the 
bounds of |HT10) that require databases to be of size polynomial in |C| 
before giving relatively-useful mechanisms. 

Theorem 5. For any constant S and any query class C , there is an 
(77, S) -relatively useful mechanism that preserves a-differential privacy for 
any database of size at least: 



\D\\i > a 



f FSD2^/.,{C) logn \ 



Proof (Proof of Theorem). Recall that the domain V of our instantiation 
of the exponential mechanism consists of all databases D' £ N" with 
|-D'||i = m with m = O ^-^^ ■ log^ (^))-'^ particular, by corollary [T] 



there exists a D* G V such that: 



q{D)-^-l^q{D') <7j\\D\ 



If ||-D||i is not public knowledge, it can be estimated to small constant error using the 
Laplace mechanism IDMNS06] . losing only additive constants in the approximation 
parameter e and privacy parameter a. This does not affect our results. 
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By the definition of our mechanism, such a D* is output witli probability 
proportional to at least: 

Similarly, any G V such that jg(D) - il|^^g(I>*)| > 2r7l|I>|li is 
output with probability proportional to at most: 

j^^~exp(-^II^JI^") 

Let Vb denote the set of all such . Because I©! = n™, we have that: 

Pr[D' = D'] ^ exp(-^^MkH) _ fv\\D\Ua 

„ ' —4 > ttt^t; ~ n ■ exp ' 

Pv[D' G Vb] - n™ • exp(-2iMkfi) 

Rearranging terms, we have: 

TD rn' ^ -n 1 ^ ™ f '^ll-^lli^^ 
Pr[_D G PbJ < n exp I — ^ — 

Solving, we find that this bad event occurs with probability at most S 
for any database D with: 

, „ , , „ f "/n log n , 1 

D 1 > ^ ^ + log- 

\ ria 

We remark that the above mechanism is the analogue of the general 
release mechanism of [BLR08) . and answers linear queries in the ojflme 
setting, when all queries C are known to the mechanism in advance. This 
is not necessary, however. In the same way as above, corollary [1] can also 
be used to generalize the Median Mechanism of Roth and Roughgarden 
[RRlOj . to achieve roughly the same bounds, but in the online setting, 
in which queries arrive online, and the mechanism must privately answer 
queries as they arrive, without knowledge of future queries. This results 
in the following theorem: 

Theorem 6. There exists a mechanism such that for every sequence of 
adaptively chosen queries qi,q2, ■ ■ ■ arriving online, chosen from some 
(possibly infinite) set C (unknown to the mechanism), the mechanism is 
{rj,S) useful with respect to C and preserves {a, r)- differential privac^ 
where t is a negligible function of n, for any database D with size at 
least: 



\D\\i > n 



\ arf' 



3 



This is an approximate form of differential privacy. Specifically, a mechanism M : 
R" — >■ R" is (a, r)-differentially private, if for any two databases D\,D2 such that 
lid - -D2II1 < 1, and for any S C R!^: 

Pr[M(Di) £S]<e" Pr[M(D2) G S] + r 
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Remark 1. Notice that for finite classes of linear queries, we may replace 
the fat shattering dimension in the bounds of both theorems [5] and |6] 
with log |C| if we so choose. 

5 Conclusion 

In this paper, we have generalized the techniques used by Blum Ligett 
and Roth, [BLROSj and Roth and Roughgarden |RR10] from the class of 
predicate queries to the more general class of linear queries. This gives 
the first mechanism for answering every linear query from some class C 
with noise that is bounded by a parameter other than the cardinality of 
C; in particular, we have given the first mechanism for answering all of 
the linear queries in certain infinite classes of queries beyond predicate 
queries. We have shown that the relevant parameter is the Fat-Shattering 
dimension of the class, which is a generalization of VC-dimension to non- 
boolean valued queries. In particular (ignoring other parameters), it is 
necessary and sufficient to add noise proportional to the fat shattering 
dimension of C. Our results show, among other things, that the sample 
complexity needed to privately agnostically learn real valued functions is 
polynomially related to the sample complexity needed to non-privately 
agnostically learn real valued functions. 

At a high level, the same technique can be applied for any class of queries, 
all of the answers to which can be summarized by some 'small' object. 
It is then sufficient to instantiate the exponential mechanism only over 
this much smaller set of objects (rather than the set of all databases) 
to obtain a useful mechanism. In the case of linear queries, we have 
shown that the answers to many queries can be summarized by integer 
valued databases with small l\ norm. An interesting future direction is 
to determine what types of nonlinear (but low sensitivity) queries have 
similar small summarizes from which useful mechanisms can be derived. 
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